Add spanmetricsconnector for RED metrics from Datadog traces - #3044

The datadogreceiver contrib component is already compiled in and
documented (README's "Ingesting Datadog traces, metrics, and logs")
for accepting Datadog Agent traces, metrics, and logs directly into
HyperDX. There's no way to turn those ingested traces into span-derived
RED metrics (call counts, error rates, latency) once they land, since
spanmetricsconnector isn't part of the build.

This is a common downstream need for anyone using Datadog Agent traces
today: the same span data commonly needs both a "calls" counter and a
duration histogram broken out by service/operation/status, computed
once at the collector rather than per-dashboard-query. Latency
specifically benefits from spanmetricsconnector's exponential
histogram support (histogram.exponential.max_size), which HyperDX's
ClickHouse schema already stores natively via the dedicated
otel_metrics_exponential_histogram table - no schema changes needed on
that side.

Purely additive and opt-in, like the existing routingconnector: being
compiled in adds no runtime behavior on its own, only availability for
a user's own pipeline config (CUSTOM_OTELCOL_CONFIG_FILE in OpAMP mode,
or a standalone config) to wire it in, the same pattern the README
already documents for datadogreceiver itself ("add a datadog receiver
block to your collector config").

Verified against this repo's own documented build process (README's
"Step 3: Verify"): `docker build -f docker/otel-collector/Dockerfile
--target dev` succeeds, and the resulting binary's `components` output
confirms spanmetricsconnector v0.155.0 is compiled in.

The README's "Included components" section is the canonical 1:1
inventory of every compiled component, but the Connectors table
only listed forward and routing, leaving the newly-compiled
spanmetricsconnector undiscoverable via the doc that's supposed to
advertise it.

Verified against opampController.ts and config.yaml's own comment: in
OpAMP supervisor mode, receivers/exporters on the default traces/metrics
pipelines are set dynamically and overwrite (not merge with) whatever
CUSTOM_OTELCOL_CONFIG_FILE adds there - the same mechanism already
documented for why processors: is kept out of the remote config. Adding
spanmetrics to those pipelines in OpAMP mode is silently dropped, no
error. Documented the working pattern (separate, non-colliding pipeline
names) for OpAMP mode, alongside the direct approach that does work in
standalone mode.

Three real bugs in the examples added last commit, all confirmed by
running otelcontribcol validate --config against the actual compiled
binary rather than just fixing on faith:

- Missing top-level `connectors:` block. A connector referenced only in
  a pipeline's receivers/exporters fails to start:
  "service::pipelines::traces: references exporter \"spanmetrics\"
  which is not configured" - reproduced verbatim.
- OpAMP example referenced the `datadog` receiver unconditionally, but
  it only exists when ENABLE_DATADOG_RECEIVER=true. Reproduced the
  exact failure this causes too: an undefined receiver fails the
  entire config's validation, not just the pipeline that names it.
- Missing `processors: [memory_limiter, batch]`, the convention every
  other user-added pipeline in this repo follows.

Also promoted the section from a Datadog-nested `###` to its own `##`
- spanmetrics works with any trace source, not just Datadog, and was
  unfindable for OTLP-only users nested under the Datadog auth
  subsection.

Both corrected shapes pass `otelcontribcol validate --config` cleanly.

Three more real issues, all verified against the actual compiled
binary rather than fixed on faith:

- spanmetrics aggregates unbounded by default, and memory_limiter
  doesn't protect its internal dimension map (that only gates incoming
  pipeline data, not a connector's own separate state) - on a
  Datadog-firehose source, high-cardinality span names would grow that
  map until OOM regardless of memory_limiter's own setting. Added
  aggregation_cardinality_limit to both examples. Confirmed the field
  name is real (not guessed) by validating it against the compiled
  binary, and separately confirmed `validate` does real strict field
  checking by watching it reject a made-up field name.

- With no `namespace`, the connector emits bare `calls`/`duration`
  metric names into the same shared otel_metrics_sum/otel_metrics_histogram
  tables every other source writes to, keyed by MetricName - a real
  collision risk. Added `namespace: traces.span.metrics` to both
  examples, matching the naming convention already reverse-engineered
  from a real Chronosphere gateway config.

- The claim that processors: [memory_limiter, batch] matters on all
  four example pipelines didn't hold: standalone traces/metrics
  already inherit it from the base config (re-declaring is a no-op,
  per this README's own "only re-declare what you're retuning"
  convention), and OpAMP's traces/spanmetrics has no ClickHouse
  exporter, so batching does nothing there. Removed the redundant
  re-declaration from the standalone example and rescoped the prose to
  what's actually true. Verified the standalone example still merges
  and validates correctly without the redundant processors line, using
  the same multi --config invocation the real container uses (base
  config.yaml + standalone-config.yaml + this example).

- Standalone example: re-list `datadog` in both traces/metrics receiver
  lists, since confmap replaces pipeline lists wholesale rather than
  appending - previously silently dropped Datadog metric ingestion for
  anyone following the earlier datadogreceiver section too. Verified
  fixed shape validates cleanly via otelcontribcol validate --config
  (exit 0, with datadog receiver actually defined).
- Add metrics_expiration: 5m alongside aggregation_cardinality_limit in
  both examples - the connector never expires aggregations by default,
  so a cap alone converts an OOM risk into permanent silent data loss
  for new series once a burst fills the map. Verified as a real field
  via validate (exit 0).
- Drop `batch` from OpAMP example's traces/spanmetrics pipeline (keep
  memory_limiter only) - that pipeline's only exporter is the connector
  itself, not network I/O, so batching adds latency/memory for no
  benefit. Verified still validates cleanly with batch removed.
- Changeset: patch -> minor, matching the datadogreceiver precedent
  (CHANGELOG.md 2.30.0 - a compiled-in opt-in component addition is a
  minor bump).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…rics examples

Both spanmetrics examples referenced `datadog` unconditionally in a
receiver list, guarded only by a comment - copying either example
verbatim without separately enabling/configuring Datadog broke config
validation entirely (`references receiver "datadog" which is not
configured`, failing every pipeline, not just the one referencing it).
Removed `datadog` from both executable examples and moved the
conditional-add guidance into a comment on the line it actually
applies to. Verified both corrected examples validate cleanly (exit 0)
against the real compiled binary, using the same multi --config merge
order the container actually uses (bootstrap + standalone-config +
custom for the standalone case; bootstrap + custom + a reproduction of
opampController.ts's real remote-config shape for the OpAMP case).

Also re-verified the disputed "silently dropped, no error" OpAMP
pitfall claim by building that exact bootstrap+custom+remote merge
with spanmetrics wired into the default traces/metrics pipelines: it
validates with exit 0, and a separate control test (a connector used
as an exporter but never as a receiver) does produce a real error
(`connector "spanmetrics" used as exporter in [traces] pipeline but
not used in any supported receiver pipeline`), confirming validate
does perform genuine connector-usage checks and that the merged,
fully-unreferenced case is a real silent no-op, not a validation gap.
No prose change needed there.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Verified each claim against the pinned v0.155.0 connector source
(github.com/open-telemetry/opentelemetry-collector-contrib, tag
connector/spanmetricsconnector/v0.155.0) and the real compiled binary,
not just the review's say-so:

- Use `span_metrics`, not `spanmetrics`, as the connector name in new
  examples. Confirmed via `internal/metadata/generated_status.go`:
  Type = "span_metrics" is canonical, DeprecatedType = "spanmetrics" is
  a legacy alias (still works, registered as such in `components`
  output) - new docs shouldn't teach the deprecated name.
- Fixed the `namespace` rationale: confirmed via factory.go that
  `DefaultNamespace = "traces.span.metrics"` is already the default,
  so omitting it does NOT produce bare `calls`/`duration` names as
  previously claimed. Reworded to "pins the default for explicitness."
- Replaced `metrics_expiration` with `series_expiration` for the
  cardinality-cap pairing. Confirmed via config.go doc comments and
  connector.go's ExpireSeries call sites: `metrics_expiration` only
  retires a whole metric object once no spans arrive at all for that
  service+operation; `series_expiration` is the one that evicts
  individual stale dimension combinations from the cardinality-bounded
  map while the metric stays otherwise active - which is what the
  described cardinality-burst scenario actually needs.
- Added `aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA`.
  Cumulative (the connector default) deletes a series' accumulator on
  expiration, so its next occurrence restarts the `calls` counter from
  1 - a real counter reset that HyperDX's own cumulative-rate SQL
  (packages/common-utils/src/core/renderChartConfig.ts, confirmed by
  reading it: `greatest(Value - lagInFrame(Value), 0)`) doesn't detect
  as a reset, it just clamps the delta to 0. For a low-traffic series
  that keeps expiring between spans, every value stays 1 and every
  computed delta is 0 forever - a real "increase" chart under-report,
  not a hypothetical one. Delta temporality passes Value straight
  through with no diffing, sidestepping this entirely, and matches
  Chronosphere's own real-world gateway config.
- Added `histogram.exponential.max_size: 160` to both examples -
  confirmed via connector.go's initHistogramMetrics that the default
  (no `histogram.exponential` set) is an *explicit*-bucket histogram,
  not the exponential one the PR description's ClickHouse
  otel_metrics_exponential_histogram claim depends on. Without this
  field the examples didn't actually deliver the benefit the PR
  describes.
- Broadened the OpAMP example's `datadog` guard comment: it previously
  said "if and only if ENABLE_DATADOG_RECEIVER=true", which is wrong
  for a user who defines their own `datadog` receiver via
  CUSTOM_OTELCOL_CONFIG_FILE instead of the env var - confmap merges
  receiver maps key-by-key regardless of how a key got there.

All five fields verified together against the real compiled binary via
`otelcontribcol validate --config` (both standalone and OpAMP merge
shapes, exit 0). The standalone shape was also actually started (not
just validated): reaches "Everything is ready. Begin running and
processing data." with the span_metrics connector live.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ics section

- Standalone example put the conditional `datadog` guard on
  `metrics: receivers:`, but Datadog *trace* data only reaches
  `span_metrics` via `traces: receivers:` (span_metrics is an exporter
  on the traces pipeline, not metrics). Adding `datadog` to
  metrics.receivers only affects Datadog's own native-metrics ingestion,
  unrelated to span-derived metrics - moved the guard and made
  `traces:` set `receivers:` explicitly instead of leaving it implicit.
  Verified both the bare example (exit 0) and the same shape with a
  real `datadog` receiver defined and added to `traces.receivers`
  (exit 0) against the compiled binary.
- Reworded the "Datadog traces have no other way to produce these"
  claim - HyperDX already computes RED metrics at query time from the
  traces table for any trace source (packages/app/src/serviceDashboard.ts's
  getExpressions/isSpanKindServer). The real motivation is precomputing
  once at the collector plus the exponential-histogram latency series.
- Updated the Connectors summary table's `spanmetrics` row to
  `span_metrics` for consistency with the rest of the section.
- Dropped the two "Chronosphere's own gateway collector" appeals
  (including a dangling ", again" referencing an unshown review
  thread) - both claims already stand on the in-repo/upstream
  reasoning stated right before them.

Also re-verified two disputed bot claims from this same review round
against a freshly rebuilt image before touching anything:
- "`span_metrics` is not a valid connector type, there's no aliasing
  mechanism" - false. `components` output lists `span_metrics` as the
  registered name; `validate --config` accepts it (error messages for
  a bad field even correctly attribute it to
  `'spanmetricsconnector.Config'`); the connector's own factory.go
  calls `xconnector.WithDeprecatedTypeAlias(metadata.DeprecatedType)`,
  which is an explicit aliasing mechanism registering `spanmetrics` as
  a working deprecated alias alongside the canonical `span_metrics`.
  Verified both names validate successfully independently.
- "`series_expiration` is not a real field" - false. Present in the
  pinned v0.155.0 config.go with its own doc comment and validation
  branch, used in connector.go via `ExpireSeries()`; validate accepts
  it, and a deliberate typo of it (along with another made-up field)
  is correctly rejected in the same error message, confirming
  validate's key-checking is real and this field passes because it
  exists, not because validation is lax.
No doc changes made for either - the existing content was already
correct.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>